This will address #875

I think will need to add to doc about this and also, whether it would be appropriate to hardcode the train/test split or allow more open control?

I think will need to add to doc about this and also, whether it would be appropriate to hardcode the train/test split or allow more open control?

@NanoCode012 What do you mean about hardcoding?

Do you have an example or documentation of how this can be used?

I was trying to reserve engineer how it's supposed to work, and maybe there's a bug here:

https://github.com/OpenAccess-AI-Collective/axolotl/blob/18f811978c01d567c2294140f53abcf8c086e337/src/axolotl/utils/data.py#L443

    dataset, prompters = load_tokenized_prepared_datasets(
        tokenizer, cfg, default_dataset_prepared_path
    )

Shouldn't you pass split=split to load_tokenized_prepared_datasets? Otherwise it will never load from cfg.test_datasets.